<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Getting started &mdash; SIMULATeQCD 0.7 documentation</title>
      <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
      <link rel="stylesheet" href="../_static/togglebutton.css" type="text/css" />
      <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/togglebutton.js"></script>
        <script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Contributions" href="../02_contributions/contributions.html" />
    <link rel="prev" title="SIMULATeQCD" href="../index.html" /> 
</head>

<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #343131" >
            <a href="../index.html" class="icon icon-home"> SIMULATeQCD
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Getting started</a></li>
<li class="toctree-l1"><a class="reference internal" href="../02_contributions/contributions.html">Contributions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../03_applications/applications.html">Applications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../04_codeBase/codeBase.html">Code base</a></li>
<li class="toctree-l1"><a class="reference internal" href="../05_modules/modules.html">Modules</a></li>
</ul>

        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu"  style="background: #343131" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">SIMULATeQCD</a>
      </nav>

      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
      <li>Getting started</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/01_gettingStarted/gettingStarted.md.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
             
  <section class="tex2jax_ignore mathjax_ignore" id="getting-started">
<h1>Getting started<a class="headerlink" href="#getting-started" title="Permalink to this headline"></a></h1>
<p>There are two possible ways to build SIMULATeQCD. If you are running on your own laptop or desktop and have an NVIDIA GPU,
we recommend that you use the container build. The container will automatically grab all software you need.
If you are running on an HPC system or want to use AMD, we recommmend you compile manually and ensure that all needed
software already exists on the system you’re using.</p>
<section id="prerequisites">
<h2>Prerequisites<a class="headerlink" href="#prerequisites" title="Permalink to this headline"></a></h2>
<p>Before cloning anything, we recommend you get <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code>. The reason we recommend this is that we have several configurations
used to test some of the SIMULATeQCD methods. These are too large to keep in a conventional repository, so we host them
using <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code>. SIMULATeQCD will work without it, but you won’t be able to run the tests.</p>
<p>You can install <a class="reference external" href="https://git-lfs.github.com/">git-lfs</a> by</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># For Debian-based system</span>
sudo apt install git-lfs
 
<span class="c1"># For Arch-based system</span>
sudo pacman -S git-lfs
</pre></div>
</div>
<p>and activate it by calling <code class="docutils literal notranslate"><span class="pre">git</span> <span class="pre">lfs</span> <span class="pre">install</span></code>. If you do not have superuser privileges where you are, you can use <a class="reference external" href="https://www.gnu.org/software/wget/">wget</a> as follows:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>wget https://github.com/git-lfs/git-lfs/releases/download/v3.0.2/git-lfs-linux-amd64-v3.0.2.tar.gz
tar -xf git-lfs-linux-amd64-v3.0.2.tar.gz
<span class="nv">PREFIX</span><span class="o">=</span>/path/to/install/dir ./install.sh
</pre></div>
</div>
<p>and you can activate it with <code class="docutils literal notranslate"><span class="pre">/path/to/install/dir/bin/git-lfs</span> <span class="pre">install</span></code>. Finally you will need to add <code class="docutils literal notranslate"><span class="pre">export</span> <span class="pre">PATH=/path/to/install/dir/bin:$PATH</span></code> to your <code class="docutils literal notranslate"><span class="pre">.bashrc</span></code>.</p>
</section>
<section id="how-to-download-the-code">
<h2>How to download the code<a class="headerlink" href="#how-to-download-the-code" title="Permalink to this headline"></a></h2>
<p>First, make sure you have activated git-lfs using <code class="docutils literal notranslate"><span class="pre">git</span> <span class="pre">lfs</span> <span class="pre">install</span></code>.
The code can then be cloned to your folder using:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/LatticeQCD/SIMULATeQCD.git
</pre></div>
</div>
<p>If you are using two-factor authentication on GitHub, which is likely the case if you
are interested in developing SIMULATeQCD, you will need to use the command</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:LatticeQCD/SIMULATeQCD.git
</pre></div>
</div>
</section>
<section id="build-manual">
<h2>Build (manual)<a class="headerlink" href="#build-manual" title="Permalink to this headline"></a></h2>
<p>If you would like to use AMD, would like to make substantial contributions to SIMULATeQCD,
or are running on an HPC system, you will need to do a manual build.
If you have an NVIDIA GPU, are running locally, and don’t plan to do much development, we recommend you
skip down to the <a class="reference internal" href="#build-container"><span class="std std-doc">container build</span></a> section, because the container will take
care of all the hassle of finding and installing software automatically.</p>
<p>For the manual build, you will need to make sure the following are installed:</p>
<ol class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">cmake</span></code> (Some versions have the “–pthread” compiler bug. Versions that definitely work are <a class="reference external" href="https://gitlab.kitware.com/cmake/cmake/tree/v3.14.6">3.14.6</a> or <a class="reference external" href="https://gitlab.kitware.com/cmake/cmake/-/tree/v3.19.2?ref_type=tags">3.19.2</a>.)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">C++</span></code> compiler with <code class="docutils literal notranslate"><span class="pre">C++17</span></code> support.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">MPI</span></code> (e.g. <code class="docutils literal notranslate"><span class="pre">openmpi-4.0.4</span></code>).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CUDA</span> <span class="pre">Toolkit</span></code> version 11+ or <code class="docutils literal notranslate"><span class="pre">HIP</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">-r</span> <span class="pre">requirements.txt</span></code> to build the documentation.</p></li>
</ol>
<section id="building-source-with-cuda">
<h3>Building source with CUDA<a class="headerlink" href="#building-source-with-cuda" title="Permalink to this headline"></a></h3>
<p>To build the source with CUDA, you need to have the <code class="docutils literal notranslate"><span class="pre">CUDA</span> <span class="pre">Toolkit</span></code> version 11.0 or higher installed on your machine.
To setup the compilation, create a folder outside of the code directory (e.g. <code class="docutils literal notranslate"><span class="pre">../buildSIMULATeQCD/</span></code>) and <strong>from there</strong> call the following example script:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cmake ../SIMULATeQCD/ <span class="se">\</span>
-DARCHITECTURE<span class="o">=</span><span class="s2">&quot;80&quot;</span> <span class="se">\</span>
-DUSE_GPU_AWARE_MPI<span class="o">=</span>ON <span class="se">\</span>
-DUSE_GPU_P2P<span class="o">=</span>ON <span class="se">\</span>
</pre></div>
</div>
<p>Here, it is assumed that your source code folder is called <code class="docutils literal notranslate"><span class="pre">SIMULATeQCD</span></code>. <strong>Do NOT compile your code in the source code folder!</strong>
You can set the CUDA installation path manually by setting the <code class="docutils literal notranslate"><span class="pre">cmake</span></code> parameter <code class="docutils literal notranslate"><span class="pre">-DCUDA_TOOLKIT_ROOT_DIR</span></code>.
<code class="docutils literal notranslate"><span class="pre">-DARCHITECTURE</span></code> sets the GPU architecture (i.e. <a class="reference external" href="https://en.wikipedia.org/wiki/CUDA#GPUs_supported">compute capability</a> version without the decimal point). For example use “70” for Volta or “80” for Ampere.</p>
</section>
<section id="building-source-with-hip-for-nvidia-platforms-experimental">
<h3>Building source with HIP for NVIDIA platforms (Experimental!)<a class="headerlink" href="#building-source-with-hip-for-nvidia-platforms-experimental" title="Permalink to this headline"></a></h3>
<p>In order to build the source with HIP for NVIDIA platforms,
you need to make sure that</p>
<ul class="simple">
<li><p>HIP is properly installed on your machine</p></li>
<li><p>CUDA is properly installed on your machine</p></li>
<li><p>The environment variable <code class="docutils literal notranslate"><span class="pre">HIP_PATH</span></code> holds the path to the HIP installation folder</p></li>
<li><p>The environment variables <code class="docutils literal notranslate"><span class="pre">CC</span></code> and <code class="docutils literal notranslate"><span class="pre">CXX</span></code> hold the path to the HIP clang compiler</p></li>
</ul>
<p>To setup the compilation, create a folder outside of the code directory (e.g. <code class="docutils literal notranslate"><span class="pre">../buildSIMULATeQCD/</span></code>) and <strong>from there</strong> call the following example script:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cmake ../SIMULATeQCD/ <span class="se">\</span>
-DARCHITECTURE<span class="o">=</span><span class="s2">&quot;80&quot;</span> <span class="se">\</span>
-DUSE_GPU_AWARE_MPI<span class="o">=</span>ON <span class="se">\</span>
-DUSE_GPU_P2P<span class="o">=</span>OFF <span class="se">\</span>
-DBACKEND<span class="o">=</span><span class="s2">&quot;hip_nvidia&quot;</span> <span class="se">\</span>
</pre></div>
</div>
<p>Here, it is assumed that your source code folder is called <code class="docutils literal notranslate"><span class="pre">SIMULATeQCD</span></code>. <strong>Do NOT compile your code in the source code folder!</strong>
You can set the HIP installation path manually by setting the <code class="docutils literal notranslate"><span class="pre">cmake</span></code> parameter <code class="docutils literal notranslate"><span class="pre">-DHIP_PATH</span></code>.
You can also set the CUDA installation path manually by setting the <code class="docutils literal notranslate"><span class="pre">cmake</span></code> parameter <code class="docutils literal notranslate"><span class="pre">-DCUDA_TOOLKIT_ROOT_DIR</span></code>.
<code class="docutils literal notranslate"><span class="pre">-DARCHITECTURE</span></code> sets the GPU architecture.
<code class="docutils literal notranslate"><span class="pre">-DUSE_GPU_P2P=ON</span></code> is not yet supported by this backend.</p>
</section>
<section id="building-source-with-hip-for-amd-platforms-experimental">
<h3>Building source with HIP for AMD platforms (Experimental!)<a class="headerlink" href="#building-source-with-hip-for-amd-platforms-experimental" title="Permalink to this headline"></a></h3>
<p>In order to build the source with HIP for AMD platforms,
you need to make sure that</p>
<ul class="simple">
<li><p>HIP is properly installed on your machine</p></li>
<li><p>The environment variable <code class="docutils literal notranslate"><span class="pre">HIP_PATH</span></code> holds the path to the HIP installation folder</p></li>
<li><p>The environment variables <code class="docutils literal notranslate"><span class="pre">CC</span></code> and <code class="docutils literal notranslate"><span class="pre">CXX</span></code> hold the path to the HIP clang compiler</p></li>
</ul>
<p>To setup the compilation, create a folder outside of the code directory (e.g. <code class="docutils literal notranslate"><span class="pre">../buildSIMULATeQCD/</span></code>) and <strong>from there</strong> call the following example script:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cmake ../SIMULATeQCD/ <span class="se">\</span>
-DARCHITECTURE<span class="o">=</span><span class="s2">&quot;gfx906,gfx908&quot;</span> <span class="se">\</span>
-DUSE_GPU_AWARE_MPI<span class="o">=</span>ON <span class="se">\</span>
-DUSE_GPU_P2P<span class="o">=</span>OFF <span class="se">\</span>
-DBACKEND<span class="o">=</span><span class="s2">&quot;hip_amd&quot;</span> <span class="se">\</span>
</pre></div>
</div>
<p>Here, it is assumed that your source code folder is called <code class="docutils literal notranslate"><span class="pre">SIMULATeQCD</span></code>. <strong>Do NOT compile your code in the source code folder!</strong>
You can set the HIP installation path manually by setting the <code class="docutils literal notranslate"><span class="pre">cmake</span></code> parameter <code class="docutils literal notranslate"><span class="pre">-DHIP_PATH</span></code>.
<code class="docutils literal notranslate"><span class="pre">-DARCHITECTURE</span></code> sets the GPU architecture. For example gfx906,gfx908.
<code class="docutils literal notranslate"><span class="pre">-DUSE_GPU_P2P=ON</span></code> is not yet supported by this backend.</p>
</section>
<section id="compiling-particular-executables">
<h3>Compiling particular executables<a class="headerlink" href="#compiling-particular-executables" title="Permalink to this headline"></a></h3>
<p>Inside the build folder, you can now begin to use <code class="docutils literal notranslate"><span class="pre">make</span></code> to compile your executables, e.g.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>make NameOfExecutable
</pre></div>
</div>
<p>If you would like to speed up the compiling process, add the option <code class="docutils literal notranslate"><span class="pre">-j</span></code>, which will compile in parallel using all available CPU threads. You can also specify the number of threads manually using, for example, <code class="docutils literal notranslate"><span class="pre">-j</span> <span class="pre">4</span></code>.</p>
<p>You also have the option to compile certain subsets of executables. For instance <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">tests</span></code> will make all the executables used for testing.
One can also compile <code class="docutils literal notranslate"><span class="pre">applications</span></code>, <code class="docutils literal notranslate"><span class="pre">examples</span></code>, <code class="docutils literal notranslate"><span class="pre">profilers</span></code>, <code class="docutils literal notranslate"><span class="pre">tools</span></code>, and <code class="docutils literal notranslate"><span class="pre">everything</span></code>. To see a full list of available executables,
look at <code class="docutils literal notranslate"><span class="pre">SIMULATeQCD/CMakeLists.txt</span></code>.</p>
</section>
</section>
<section id="build-container">
<h2>Build (container)<a class="headerlink" href="#build-container" title="Permalink to this headline"></a></h2>
<p>If you just want to get something running quickly on your laptop or desktop, this is likely the easiest way to go.</p>
<section id="install-podman">
<h3>Install Podman<a class="headerlink" href="#install-podman" title="Permalink to this headline"></a></h3>
<section id="on-rhel-based-rocky-centos-rhel-systems">
<h4>On RHEL-based (Rocky/CentOS/RHEL) systems<a class="headerlink" href="#on-rhel-based-rocky-centos-rhel-systems" title="Permalink to this headline"></a></h4>
<p>Before continuing make sure there are no updates pending with <code class="docutils literal notranslate"><span class="pre">sudo</span> <span class="pre">dnf</span> <span class="pre">update</span> <span class="pre">-y</span> <span class="pre">&amp;&amp;</span> <span class="pre">sudo</span> <span class="pre">dnf</span> <span class="pre">install</span> <span class="pre">-y</span> <span class="pre">podman</span></code> and then reboot with <code class="docutils literal notranslate"><span class="pre">sudo</span> <span class="pre">reboot</span></code>. (The reboot just makes avoiding permissions/kernel issues easy because that stuff is reread on boot.)</p>
</section>
<section id="on-arch-based-systems">
<h4>On Arch-based systems<a class="headerlink" href="#on-arch-based-systems" title="Permalink to this headline"></a></h4>
<p>See <a class="reference external" href="https://wiki.archlinux.org/title/Podman">install instructions</a>. If you have installed Arch before the upgrade to shadow (as in /etc/shadow) 4.11.1-3 rootless podman may encounter some issues. The build script will check for these anomalies and prompt you if you need to fix them.</p>
</section>
<section id="other-nix-systems">
<h4>Other *NIX Systems<a class="headerlink" href="#other-nix-systems" title="Permalink to this headline"></a></h4>
<p>If you have a non RHEL-based OS see <a class="reference external" href="https://podman.io/getting-started/installation.html#linux-distributions">here</a> for installation instructions.</p>
</section>
</section>
<section id="make-sure-podman-works">
<h3>Make sure Podman works<a class="headerlink" href="#make-sure-podman-works" title="Permalink to this headline"></a></h3>
<p>Run <code class="docutils literal notranslate"><span class="pre">podman</span> <span class="pre">run</span> <span class="pre">hello-world</span></code> as your user to test your privileges.
If this does not run correctly, the container build will not function.</p>
<p>If you see the error:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ERRO</span><span class="p">[</span><span class="mi">0014</span><span class="p">]</span> <span class="n">cannot</span> <span class="n">find</span> <span class="n">UID</span><span class="o">/</span><span class="n">GID</span> <span class="k">for</span> <span class="n">user</span> <span class="n">u6042105</span><span class="p">:</span> <span class="n">No</span> <span class="n">subuid</span> <span class="n">ranges</span> <span class="n">found</span> <span class="k">for</span> <span class="n">user</span> <span class="s2">&quot;u6042105&quot;</span> <span class="ow">in</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">subuid</span> <span class="o">-</span> <span class="n">check</span> <span class="n">rootless</span> <span class="n">mode</span> <span class="ow">in</span> <span class="n">man</span> <span class="n">pages</span><span class="o">.</span>
</pre></div>
</div>
<p>this indicates someone has modified the standard user privileges or you are running an older operating system. To fix this error run <code class="docutils literal notranslate"><span class="pre">sudo</span> <span class="pre">usermod</span> <span class="pre">--add-subuids</span> <span class="pre">100000-165535</span> <span class="pre">--add-subgids</span> <span class="pre">100000-165535</span> <span class="pre">&lt;YOUR_USER&gt;</span> <span class="pre">&amp;&amp;</span> <span class="pre">podman</span> <span class="pre">system</span> <span class="pre">migrate</span></code></p>
<p><strong>WARNING</strong>: If you are SSH’ing to your server, make sure you ssh as a user and <strong>not</strong> root. If you SSH as root and then <code class="docutils literal notranslate"><span class="pre">su</span></code> to user, podman will issue <code class="docutils literal notranslate"><span class="pre">ERRO[0000]</span> <span class="pre">XDG_RUNTIME_DIR</span> <span class="pre">directory</span> <span class="pre">&quot;/run/user/0&quot;</span> <span class="pre">is</span> <span class="pre">not</span> <span class="pre">owned</span> <span class="pre">by</span> <span class="pre">the</span> <span class="pre">current</span> <span class="pre">user</span></code>. This happens because the user that originally setup <code class="docutils literal notranslate"><span class="pre">/run</span></code> is root rather than your user.</p>
</section>
<section id="build-the-code">
<h3>Build the code<a class="headerlink" href="#build-the-code" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>Update <code class="docutils literal notranslate"><span class="pre">podman-build/config.yml</span></code> with any settings you would like to use for your build. This includes your target output directory.</p>
<ol class="simple">
<li><p>You can run <code class="docutils literal notranslate"><span class="pre">&lt;where_you_downloaded&gt;/simulate_qcd.sh</span> <span class="pre">list</span></code> to get a list of possible build targets.</p></li>
<li><p>If you want to change where the code outputs to, you need to update OUTPUT_DIRECTORY in <code class="docutils literal notranslate"><span class="pre">podman-build/config.yml</span></code>. It will create a folder called build in the specified folder.</p></li>
</ol>
</li>
<li><p>Run <code class="docutils literal notranslate"><span class="pre">chmod</span> <span class="pre">+x</span> <span class="pre">./simulate_qcd.sh</span> <span class="pre">&amp;&amp;</span> <span class="pre">./simulate_qcd.sh</span> <span class="pre">build</span></code></p></li>
</ol>
</section>
</section>
<section id="how-to-run">
<h2>How to run<a class="headerlink" href="#how-to-run" title="Permalink to this headline"></a></h2>
<section id="on-a-cluster-using-slurm">
<h3>On a cluster using <code class="docutils literal notranslate"><span class="pre">slurm</span></code><a class="headerlink" href="#on-a-cluster-using-slurm" title="Permalink to this headline"></a></h3>
<p>If you are on a cluster that uses slurm, then inside of your sbatch script do not use <code class="docutils literal notranslate"><span class="pre">mpiexec</span></code> or <code class="docutils literal notranslate"><span class="pre">mpirun</span></code>, but instead do</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>srun -n &lt;NoGPUs&gt; ./&lt;program&gt;
</pre></div>
</div>
<p>where <code class="docutils literal notranslate"><span class="pre">&lt;NoGPUs&gt;</span></code> is the number of GPUs you want to use.
The <code class="docutils literal notranslate"><span class="pre">&lt;program&gt;</span></code> likely already points to some default parameter file. If you would like to pass your
own parameters, one way to do this is to make your own parameter file. This can then be passed
as an argument like</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>srun -n &lt;NoGPUs&gt; ./&lt;program&gt; /path/to/your/parameter/file
</pre></div>
</div>
<p>Example parameter files can be found in the <code class="docutils literal notranslate"><span class="pre">parameter</span></code> folder. We have tried to explain
what each parameter means in each <code class="docutils literal notranslate"><span class="pre">.param</span></code> file. You can learn more about how general parameters
are implemented <a class="reference internal" href="../02_contributions/inputParameter.html"><span class="doc std std-doc">here</span></a>.</p>
</section>
<section id="on-your-local-machine-desktop-laptop">
<h3>On your local machine (desktop, laptop, …)<a class="headerlink" href="#on-your-local-machine-desktop-laptop" title="Permalink to this headline"></a></h3>
<p>Unless you are using one GPU, any program has to be launched using <code class="docutils literal notranslate"><span class="pre">mpirun</span></code> or <code class="docutils literal notranslate"><span class="pre">mpiexec</span></code>.
For example:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>mpiexec -np &lt;NoGPUs&gt; ./&lt;program&gt; 
</pre></div>
</div>
<p>Simiarly as above, if you want to pass your own parameter file,</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>mpiexec -np &lt;NoGPUs&gt; ./&lt;program&gt; /path/to/your/parameter/file
</pre></div>
</div>
</section>
</section>
</section>


           </div>
          </div>
          <footer>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2021, LatticeQCD.</p>
  </div>

  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
   

</footer>
        </div>
      </div>
    </section>
  </div>
  <script>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 

</body>
</html>